GH test summary can be seen here:
https://github.com/yfyf/playos/actions/runs/11482660452#summary-31956262032

For some reason Github renders the logs in <pre> with extra empty lines, not sure why, will look into it.

Edit: it seems whitespace handling is kinda broken on the summary page, though the markdown gets rendered differently if I paste it into a Gist, for example. 🤦

So... decided to roll manually 🤷

Right. I had a go at TAP/bats for something else some time ago, and walked way from it. It always feels like it should be good, though. Maybe something hasn't clicked yet.

Help me understand the implications for our codebase. If you had done it via bats/TAP, would we have used a library that would have generated/help generate the right output? And then the tool would have transformed that to some summary? Or would we have still generated the right format of output ourselves, and only apply the tool to create the summary? Maybe the only extra tax is creating that markdown report manually?

Output in your failed test looks like a good thing! On main right now, it's really hard to see what failed at a glance.

Help me understand the implications for our codebase. If you had done it via bats/TAP, would we have used a library that would have generated/help generate the right output? And then the tool would have transformed that to some summary?

The BATS approach was:

• Wrap each e2e test (driver) $fooTest into a shell script ${fooTest}.bats containing @test "$fooTest" { $fooTest }
• Link farm all the tests into some $testDir
• Run the tests with bats --timing --recursive --report-formatter junit -o test-report.xml $testDir

Issues with this:

1. No console output while tests are running, which sucks when running e2e tests because they take a while, might be hanging, etc. No flags to change this.
2. The generated test-report.xml is broken because BATS doesn't escape the log output (could be fixed by piping the output in the wrapper files).
3. If any test fails, it will dump all of the logs, no option to suppress this.
4. Final summary printed contains only total counts.

3 + 4 means that if anything fails, there's no good way to see an overview in the console, because individual test outcomes are interspersed by the failing test logs.

If it were only for the GH summary report, I would say that yeah, BATS + post-processing the XML is probably a better option. But now it just seems like a big indirection with questionable benefits.

Log formatting is now properly preserved in the summary page:
https://github.com/dividat/playos/actions/runs/11484431866?pr=194#summary-31962190652

Rebased with main.

I think this is good to merge?

Yes.

I am just thinking that it would be also nice if the CLI summary report was re-used for integration tests as well. We have 9 of them now and the current for-loop fail-on-first-err in ./bin/test implementation is not ideal for local development. A summary table on Github would be helpful too. This could be a separate PR tho.

I am just thinking that it would be also nice if the CLI summary report was re-used for integration tests as well. We have 9 of them now and the current for-loop fail-on-first-err in ./bin/test implementation is not ideal for local development. A summary table on Github would be helpful too.

👍

This could be a separate PR tho.

👍